Methods and systems for eyes-free text entry

ABSTRACT

The present disclosure relates to a system for eyes-free text entry. The system may include a wearable device having a display and processing circuitry configured to receive a haptic input provided to a keyboard mounted on a finger of a user, the haptic input being an indication of an alphabetical letter, generate a list of candidate words based on the received haptic input, each candidate t word of the list of candidate words being associated with a probability thereof, display the generated list of candidate words to the user via the display of the wearable device, receive a selection of a particular candidate word of the list of candidate words, and append the particular candidate to a present sentence structure, wherein the keyboard has a layout based on a spatial model reflecting spatial awareness, by the user, of key locations on the finger.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 62/923,059, filed Oct. 18, 2019, the teaching of which isincorporated by reference herein in its entirety for all purposes.

BACKGROUND Field of the Disclosure

The present disclosure relates to finger-tip control of mobile devicesand communication.

Description of the Related Art

Appreciating that computing devices are tightly integrated into ourdaily living and working environments, users often require easy-to-carryand always-available input devices to interact with them in subtlemanners.

One-handed micro thumb-tip gestures, for instance, offer newopportunities for fast, subtle, and always-available interactions,particularly on devices with limited input space (e.g., wearables). Verymuch like gesturing on a trackpad, using the thumb-tip to interact withthe virtual world through the index finger is a natural method toperform input, and is made increasingly practical with rapid advances insensing technologies such as epidermal devices and interactive shintechnologies.

While micro thumb-tip gestures have been explored for many application,such as mobile information tasks (e.g., dialing numbers), text entry asan application of micro thumb-tip gestures is often overlooked, despitethe fact that text entry comprises approximately 40% of mobile activity.Moreover, using the thumb-tip for text entry on the index finger offersseveral potential benefits. First, text input can be carried out usingone hand, which may be important in mobile scenarios, where the otherhand may be occupied by a primary task. Second, text input can becarried out unobtrusively, which can be useful in social scenarios, suchas in a meeting where alternative solutions, like texting on a device(e.g., smartphone or watch) or using speech, may be sociallyinappropriate or prone to compromising privacy of a user. Third, textinput can be carried out without a requirement to visually observe thekeyboard and the keystrokes performed thereon. Such an ‘eyes-free’environment may lead to better performance than eyes-on input whilesaving screen real estate for devices.

However, despite these potential benefits, implementation of eves-freetext entry approach is challenging because of a lack of input space,missing proper haptic feedback, and lack of a flat and rigid surface onan index finger, for instance. To this end, a QWERTY keyboard can barelybe laid out on the index finger and the keys can be too small to type.Unlike a physical keyboard, typing on the index finger offers littleuseful haptic feedback to inform the user about which key was selected,making it more difficult for eyes-free typing. This is to say nothing ofcurved and soft nature of the tip of the index finger, which may impacttapping accuracy on what are already small “keys”.

Accordingly, a one-handed text entry technique designed for enablingthumb-tip tapping while addressing the above-described shortcomings isneeded.

The foregoing “Background” description is for the purpose of generallypresenting the context of the disclosure. Work of the inventors, to theextent it is described in this background section, as well as aspects ofthe description which may not otherwise qualify as prior art at the timeof filing, are neither expressly or impliedly admitted as prior artagainst the present invention.

SUMMARY

The present disclosure relates to a method and systems for eyes-freetext entry.

In an embodiment, the present disclosure further relates to a system foreyes-free text entry, comprising a wearable device having a display, andprocessing circuitry configured to receive a haptic input provided to akeyboard mounted on a finger of a user, the haptic input being anindication of an alphabetical letter determined based on a location ofthe haptic input on the keyboard, generate a list of candidate wordsbased on the received haptic input, each candidate word of the list ofcandidate words being associated with a probability thereof, display thegenerated list of candidate words to the user via the display of thewearable device, receive a selection of a particular candidate word ofthe list of candidate words, and append the particular candidate word ofthe list of candidate words corresponding to the received selection to apresent sentence structure, wherein the keyboard has a layout based on aspatial model reflecting spatial awareness, by the user, of keylocations on the finger. In an embodiment, the processing circuitry isconfigured to calculate the probability associated with each candidateword of the list of candidate words by generating a probability based onan application of the spatial model to the received haptic input, thespatial model describing a relationship between touch locations of theuser and locations of keys of the keyboard, generating a probabilitybased on an application of a language model to the received hapticinput, the language model providing probability distributions of asequence of words for a given language, and combining the generatedprobability based on the application of the spatial model and thegenerated probability based on the application of the language model togenerate the probability associated with each candidate word of the listof candidate words.

According to an embodiment, the present disclosure further relates to amethod of eyes-free text entry, comprising receiving, by processingcircuitry, a haptic input provided to a keyboard mounted on a finger ofa user, the haptic input being an indication of an alphabetical letterdetermined based on a location of the haptic input on the keyboard,generating, by the processing circuitry, a list of candidate words basedon the received haptic input, each candidate word of the list ofcandidate words being associated with a probability thereof, displaying,by the processing circuitry, the generated list of candidate words tothe user via a display of a wearable device, receiving, by theprocessing circuitry, a selection of a particular candidate word of thelist of candidate words, and appending, by the processing circuitry, theparticular candidate word of the list of candidate words correspondingto the received selection to a present sentence structure, wherein thekeyboard has a layout based on a spatial model reflecting spatialawareness, by the user, of key locations on the finger.

According to an embodiment, the present disclosure further relates to anapparatus for eyes-free text entry, comprising processing circuitryconfigured to receive a haptic input provided to a keyboard mounted on afinger of a user, the haptic input being an indication of analphabetical letter determined based on a location of the haptic inputon the keyboard, generate a list of candidate words based on thereceived haptic input, each candidate word of the list of candidatewords being associated with a probability thereof, display the generatedlist of candidate words to the user via a display of a wearable device,receive a selection of a particular candidate word of the list ofcandidate words, and append the particular candidate word of the list ofcandidate words corresponding to the received selection to a presentsentence structure, wherein the keyboard has a layout based on a spatialmodel reflecting spatial awareness, by the user, of key locations on thefinger.

The foregoing paragraphs have been provided by way of generalintroduction, and are not intended to limit the scope of the followingclaims. The described embodiments, together with further advantages,will be best understood by reference to the following detaileddescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1A is an illustration of an aspect of a system for eves-freeone-handed text entry, according to an exemplary embodiment of thepresent disclosure;

FIG. 1B is an illustration of an aspect of a system for eyes-freeone-handed text entry, according to an exemplary embodiment of thepresent disclosure;

FIG. 2 is a cartoon rendering of a keyboard implemented within a systemfor eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure;

FIG. 3A is an image of a keyboard implemented within a system foreyes-free one-handed text entry, according to an exemplary embodiment ofthe present disclosure;

FIG. 3B is an image of a keyboard implemented within a system foreyes-free one-handed text entry, according to an exemplary embodiment ofthe present disclosure;

FIG. 3C is an image of a keyboard implemented within a system foreyes-free one-handed text entry, according to an exemplary embodiment ofthe present disclosure;

FIG. 4A is an illustration of potential layouts of a keyboardimplemented within a system for eyes-free one-handed text entry,according to an exemplary embodiment of the present disclosure;

FIG. 4B is an illustration of layout of a keyboard implemented within asystem for eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure;

FIG. 5A is a flow diagram of a method of implementing a system foreyes-free one-handed text entry, according to an exemplary embodiment ofthe present disclosure;

FIG. 5B is a flow diagram of a sub process of a method of implementing asystem for eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure;

FIG. 5C is a flow diagram of a sub process of a method of implementing asystem for eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure;

FIG. 6A is an image of a user typing in front of a monitor surrounded by5 Vicon cameras via eyes-free one-handed text entry, according to anexemplary embodiment of the present disclosure;

FIG. 6B is an image of markers attached to fingers of a user, accordingto an exemplary embodiment of the present disclosure:

FIG. 6C is an image of clay models of fingers of a user that have beenused for three-dimensional scanning, according to an exemplaryembodiment of the present disclosure;

FIG. 7 is a rendering of a three-dimensional touch simulation of twointersected fingers, according to an exemplary embodiment of the presentdisclosure;

FIG. 8 is a series of scatter plots with 95% confidence ellipses oftouch points in a 26 key QWERTY keyboard layout, according to anexemplary embodiment of the present disclosure;

FIG. 9A is a scatter plot with 95% confidence ellipses of touch pointsin a keyboard layout, according to an exemplary embodiment of thepresent disclosure;

FIG. 9B is a scatter plot with 95% confidence ellipses of touch pointsin a keyboard layout, according to an exemplary embodiment of thepresent disclosure;

FIG. 9C is a scatter plot with 95% confidence ellipses of touch pointsin a keyboard layout, according to an exemplary embodiment of thepresent disclosure;

FIG. 10A is an illustration of a layout of a keyboard implemented withina system for eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure;

FIG. 10B is an illustration of a layout of a keyboard implemented withina system for eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure;

FIG. 11 is an image of a keyboard implemented within a system foreyes-free one-handed text entry, according to an exemplary embodiment ofthe present disclosure;

FIG. 12A is a graphical illustration of text entry speed across fourblocks, according to an exemplary embodiment of the present disclosure;

FIG. 12B is a graphical illustration of text entry error rate acrossfour blocks, according to an exemplary embodiment of the presentdisclosure;

FIG. 13 is a hardware configuration of a computer integrally-formed orin communication with an apparatus of a system for eyes-free one-handedtext entry, according to an exemplary embodiment of the presentdisclosure; and

FIG. 14 is a hardware configuration of an apparatus, such as a wearabledevice, having computational resources for performing methods of thesystem for eyes-free one-handed text entry, according to an exemplaryembodiment of the present disclosure.

DETAILED DESCRIPTION

The terms “a” or “an”, as used herein, are defined as one or more thanone. The term “plurality”, as used herein, is defined as two or morethan two. The term “another”, as used herein, is defined as at least asecond or more. The terms “including” and/or “having”, as used herein,are defined as comprising (i.e., open language). Reference throughoutthis document to “one embodiment”, “certain embodiments”, “anembodiment”, “an implementation”, “an example” or similar terms meansthat a particular feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodiment ofthe present disclosure. Thus, the appearances of such phrases or invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments without limitation.

There have been a number of techniques proposed to facilitate inputperformed with hand gestures and finger gestures. Various sensingapproaches have been introduced for input recognition, includingcamera-based approaches, bio-acoustic approaches, andelectromyography-based approaches, among others. Such approaches havebeen shown effective in the detection of hand gestures (e.g. fist, handwaving, finger tap on skin) and pinching motions (e.g. thumb touchingother fingers). Hand gestures can also be sensed using electricalimpedance tomography and pressure sensors on the wrist and on the arm.

A common approach for text-entry is based on gestures. For example, acontinuous stroke can be used to enter a letter or a word.Alternatively, a single letter can be entered using several discretestrokes or taps. Other commonly applied techniques include non-visualtext entry where the user has no visual access to the keyboard. However,these approaches are often focused on enabling novel text entry schemesfor visually impaired users or for touch-screen devices where screenreal-estate is considerably larger than that of a finger-tip.

Text entry on wearable devices is a considerable challenge, appreciatingthat input space is, with respect to a QWERTY keyboard with 26 keys,relatively constrained. In an effort to address this challenge, avariety of techniques, such as two-step key selection, have beenexplored. Most two-step key selection approaches, however, require theuse of two hands and use finger touch as an input modality.

Meanwhile, one-handed text entry in wearables introduces a path forward.For instance, one approach to one-handed text entry may utilize onedimensional handwriting with a unistroke gesture, while another approachmay propose a two-step typing method on a smart glass touchpad. Still,another approach may rely on thumb-to-fingers touch on a T9 keyboardmapped on finger segments providing haptic feedback.

However, the above approaches leave a gap in adequately addressingeyes-free text entry on wearable devices. Accordingly, the presentdisclosure describes: (1) a spatial model workable with thumb-tiptapping on a fingertip surface (e.g. interactive skin) and (2) anoptimized keyboard layout design for TipText. The results of a userstudy implementing TipText will then be evaluated.

As introduction, the following factors can be considered in designing aneyes-free text entry method using micro thumb-tip gestures.

First, three types of learnability can be considered: (1) learnabilityof input technique; (2) learnability of keyboard layout; and (3)learnability of eyes-free text entry. As it relates to (1) inputtechniques for text entry, several techniques including tap, directionalswipe, and whirling the wrist may be deployed. Learnability also variesamong different input techniques. For example, tapping keys is easy tolearn but swiping directionally requires more effort. In general,letter-based text entry methods require less learning effort thanword-based methods but trade-offs may exist between learnability andefficiency. For example, letter-based input methods can be slower inentering text. In view of the above, the present disclosure describeskey tapping for letter-based text entry to exploit rapid learning.Moreover, various types of keyboard designs exist, including, amongothers, those following an alphabetical order or a QWERTY layout. Withrespect to (2) the learnability of keyboard layout QWERTY is relativelyeasy to learn due to its wide adoption. Therefore, QWERTY is employedwithin the present disclosure. Further to the above, the presentdisclosure considers preserving the spatial layout of the letters tominimize learning. As it relates to (3) eyes-free typing, the adoptionof tapping and a QWERTY layout minimizes the learning curve of a newuser. It can be appreciated that, when typing in an eyes-free context,the imaginary location of a desired key, according to a user and basedon their spatial awareness, can be different from the actual location ofthe key. Thus, the user needs to learn the mapping of each key and topractice in order to develop corresponding kinesthetic memory. Throughimplementation of eyes-free typing through a system that adopts aspatial model of collected eyes-free input on the index finger, thislearning curve is lessened.

Second, and in addition to the above learnability factors, two types ofeyes-free conditions can be considered: (1) typing without looking atfinger movements and (2) typing without looking at the keyboard. Since auser input space is different from a user output space, it is importantto free the visual attention of the user from the fingers as regularlyswitching attention between where they type and where the output is mayintroduce significant cognitive overhead and lead to reducedperformance. The visual appearance of the keyboard should also beavoided since the screen, if it exists on a very small wearable device(e.g., smartwatch or head-worn display), may be very small. Therefore,screen real estate should be dedicated to the text entered by the userrather than the keyboard. Of course, on devices without a screen, theentered text can be provided via audio using a wireless headphone. Inview of the above, eyes-free text input can allow for common activitiesof daily living and mobile scenarios such as walking with the handhanging along the body. In general, precise eyes-free input ischallenging, especially on the small fingertip. This challenge can beovercome through careful design of keyboard layout, taking intoconsideration the models of both input language and natural spatialawareness of each user.

Lastly, two types of accuracy can be considered: (1) accuracy of inputtechnique and (2) accuracy of text entry method. With respect to theaccuracy of input technique (e.g., tapping precision), it can be hard toprecisely locate contact on the small input space of the index fingerbecause of the so-called “fat finger” issue. However, input does nothave to be 100% accurate. Certain amounts of tapping errors can betolerated using a statistical decoder. The efficiency of a letter-basedtext entry method is mostly related to word disambiguation. This issueappears when more than one letter is associated with an enlarged key(e.g., T9) because it is hard to tell which letter the user wants toenter. Therefore, a balance needs to be struck between key size and worddisambiguation.

According to an embodiment, and in view of the above, the presentdisclosure describes a system, apparatus, and methods for eyes-free textentry using micro thumb-tip gestures. The technique features a miniatureQWERTY keyboard residing invisibly on a first segment of an index fingerof a user. Text entry can be carried out using a thumb-tip to tap thetip of the index finger of the user. The keyboard layout can beoptimized for eyes-free text input by utilizing a spatial model of thekeyboard that reflects natural spatial awareness, by a user, of keylocations on the index finger.

The system, apparatus, and methods of the present disclosure incorporateresults of a series of user studies and computer simulated text entrytests that consider over 1,146,484 possible designs. In an embodiment,the present disclosure describes a grid with letters highly confined tothe alphabetic and spatial arrangement of a QWERTY keyboard. In anexample, the grid is a two row by three column grid including thealphabetic arrangement and spatial arrangement of the QWERTY keyboard.In a preliminary analysis, micro thumb-tip gestures, implementedaccording to the system, apparatus, and methods described herein,achieved an average text entry speed of 11.9 words per minute (WPM),with improved typing as fast as 13.3 WPM with increasing user-equipmentfamiliarity.

Referring now to the Figures, a system for eyes-free text entry usingmicro thumb-tip gestures is described. The system of the presentdisclosure, which may be referred to herein as the TipText system, andmay be considered in view of an apparatus performing similar methods,includes a one-handed text entry technique designed for enablingthumb-tip tapping on a miniature fingertip keyboard.

As shown in FIG. 1A, a TipText system 100 can include a wearable 102,such as smart glasses 107, and a fingertip keyboard positioned on anindex finger 103 of a user 101 and apposing a thumb 104 of the user 101.As shown in FIG. 1B, a TipText system 100 can include a wearable 102,such as smart watch 108, and a fingertip keyboard positioned on an indexfinger 103 of a user 101 and apposing a thumb 104 of the user 101. Foreach of FIG. 1A and FIG. 1B, the wearable 102 may be in wired orwireless communication with the fingertip keyboard positioned on theindex finger 103 of the user 101 and apposing the thumb 104 of the user101. Moreover, the wearable 102 may provide, in a realisticimplementation of the TipText system, computational resources forprocessing received haptic inputs from the user 101.

In embodiment, and with reference to either of FIG. 1A or FIG. 1B, theTipText system 100 can include a miniature QWERTY keyboard that residesinvisibly on a first segment (e.g. distal phalanx) of the index finger103 of the user 101. When typing in an eyes-free context, the user 101can interact with each “key” based on, in view of the QWERTY keyboard,natural spatial awareness of the location of the desired “key”. Thereceived haptic signal, which may include an indication of one letter ora plurality of successive letters, can then be processed by circuitry ofthe TipText system 100 and used in order to direct a corresponding wordsearch within a dictionary. The search can reveal words corresponding tothe sequence of the selected “key” or “keys” and then provide a list ofcandidate words that likely match the haptic input from the user 101.The list of candidate words can then be displayed to the user 101, via adisplay of the wearable 102, as a ranked list according to calculatedprobability of each candidate word. The calculated probability for eachcandidate word can be determined by a statistical decoder. Duringimplementation of the TipText system 100, the displayed ranked list ofcandidate words can then be evaluated by the user 101 during a wordselection mode to determine which one accurately matches the intendedword. In an embodiment, the user 101 may use swiping motions to indicatethe accuracy of a candidate word. For instance, the user 101 may swipethe thumb across the QWERTY keyboard on the index finger 103 in order toenter the selection mode. In an example, the selection mode can beentered by swiping right across the QWERTY keyboard on the index finger103. In the selection mode, the user 101 can determine whether thepresent candidate word, or in the case of the initial candidate word,the highest ranked candidate word, is the intended word. If the user 101determines the present candidate word is the intended word, the user 101can proceed with the next word in a sentence, tapping ‘keys’ of theQWERTY keyboard on the index finger 103 that corresponding to asubsequent intended word of the sentence. In an embodiment, a space willbe automatically inserted after the present candidate word becomes‘committed’ by user tapping of the subsequent intended word. If,alternatively, the user 101 determines the present candidate word is notthe intended word, the user 101 can, again, swipe across the QWERTYkeyboard on the index finger 103 in order to summon the next candidateword from the ranked list of candidate words. In an example, the nextcandidate word can be summoned by swiping right across the QWERTYkeyboard on the index finger 103.

In an embodiment, the user 101 may be able to enter the word selectionmode, and thus indicate a present haptic input session is complete, byswiping across the QWERTY keyboard on the index finger 103 in a specificmanner. This indicates haptic input related to a given word of asentence is complete.

In an embodiment, auto-complete may be implemented within the TipTextsystem 100, wherein the user 101 may be enabled to select a desired wordfrom the candidate list of words without having to enter all of theletters of the intended word.

Moreover, in an embodiment, the user 101 may be able to erase an enteredletter by swiping across the QWERTY keyboard on the index finger 103. Inan example, entered letters may be erased by swiping left across theQWERTY keyboard on the index finger 103.

According to an embodiment, the QWERTY keyboard on the index finger ofthe user may be arranged within a grid. As shown in FIG. 2 , a QWERTYkeyboard 209 may be arranged in, for instance, a three column by threecolumn grid 250 and may reside invisible on a first segment of an indexfinger 203 of a user. Though the QWERTY keyboard 209 of FIG. 2 isexemplary, it can be appreciated that a keyboard layout implementedwithin a TipText system of the present disclosure can be optimized foreyes-free input by utilizing a spatial model reflecting natural spatialawareness of a user as it relates to ‘key’ locations on the index finger203. In this way, user learning of eyes-free typing can be accelerated.

According to an embodiment, the QWERTY keyboard described with referenceto FIG. 2 can be implemented in a variety of ways, as exemplified inFIG. 3A through FIG. 3C. Generally, a QWERTY keyboard 309 can beactualized within an interactive skin 352. The interactive skin 352 maybe thin and flexible and include a contact surface 351 having a surfacearea. The surface area of the contact surface 351 may be, in an example,˜2 cm×˜2 cm, but can be generally appreciated as any area correspondingto an anticipated size and shape of a finger of a user. The contactsurface 351 of the interactive skin 352 may feature, as a grid 350thereon, a touch sensor matrix. The touch sensor matrix may be a 3×3capacitive touch sensor matrix, as an example of a tactile sensor. Thetouch sensor matrix may feature diamond shaped electrodes. In anexample, the diamond shaped electrodes may be of 5 mm diameter andarranged with 6.5 mm center-to-center spacing. Of course, the selectionof a diamond shaped electrode is merely arbitrary, as any shapedelectrode may be implemented herein. Moreover, the sizing and spacing ofthe electrodes is dictated by specific constraints of the keyboarddesign and a statistical decoder used in processing haptic input, andshould be considered dynamic across implementations.

Specifically, and as in FIG. 3A, the interactive skin 352 may be apolyethylene terephthalate film refined by a conductive inkjet printingprocess. An inkjet printer may be filled with, as the conductive ink, asilver nanoparticle ink and the electrodes of a grid 350 may begenerated therefrom. In an example, the inkjet printer may be a CanonIP100 desktop inkjet printer and the conductive silver nanoparticle inkmay be Mitsubishi NBSIJ-MU01 ink.

As in FIG. 3B, the interactive skin 352 may be a flexible printedcircuit. In an example, the flexible printed circuit measured 0.025 mmto 0.125 mm thick and 21.5 mm×27 mm wide.

As in FIG. 3C, a highly conforming version of the interactive skin 352was generated. The interactive skin 352 of FIG. 3C was fabricated ontemporary tattoo paper. Conductive traces were then screen printed ontothe temporary tattoo paper using silver ink overlaid withpoly(3,4-ethylenedioxythiphene) polystyrene sulfonate (PEDOT:PSS). Alayer of resin binder was then printed between electrode layers toelectrically isolate each one. In an embodiment, two layers of temporarytattoos can be applied in order to isolate the sensor from the skin. Inan example, the sliver ink is obtained from Gwent (ID#C2130809D5), thePEDOT:PSS is obtained from Gwent (ID#C2100629D1), and the resin binderis obtained from Gwent OW R2070613P2).

According to an embodiment, the interactive skins described above arecomponents of the TipText system and can be in electrical communicationwith processing circuitry of the TipText system. Accordingly, theprocessing circuitry of the TipText system can include, at least, atactile sensor for receiving haptic inputs from the user via theinteractive skin.

According to an embodiment, a QWERTY keyboard of the interactive skin ofthe TipText system can employ any number of arrangement combinations ofthe “keys” therein. For instance, FIG. 4A provides a non-limitingillustration of a plurality of QWERTY keyboard layouts that can beimplemented. FIG. 4A includes 16 possible keyboard layouts of a 1×5grid, 32 possible keyboard layouts of a 2×3 grid, and 2 possiblekeyboard layouts of a 3×2 grid. In an example, the exemplary QWERTYkeyboard layout of FIG. 4B is a 2×3 grid that scored highly whenevaluated by a language model.

Equipped with the above-described TipText system, a user may interactwith the TipText system according to method 510 of FIG. 5A. Steps ofmethod 510 are described as performed by processing circuitry of anapparatus that processes data signals from the interactive skin andcontrols an output to an user display. In an embodiment, the apparatusmay be a wearable device of a user. To this end, the methods describedherein may be performed by the wearable device of the user and may be asoftware application downloadable thereto.

At step 515 of method 510, user input can be received from a keyboard ofan interactive skin of the TipText system. Haptic inputs of the user canbe transduced by a tactile sensor and can be received by processingcircuitry configured to implement a statistical decoder to interpret thehaptic inputs.

In an embodiment, an in order to indicate that a haptic input session iscomplete, the user may perform a specific motion or other action on theQWERTY keyboard. For example, the specific motion may be a swipe acrossthe QWERTY keyboard and the action may indicate that the user is readyfor a word selection mode, or selection mode.

At sub process 520 of method 510, the statistical decoder can be appliedto the input received from the keyboard during the haptic input session.As will be described with reference to FIG. 5B, application of thestatistical decoder includes application of a spatial model and alanguage model to generate a list of candidate words that may be anintended word input by the user.

At step 530 of method 510, the list of candidate words generated at subprocess 520 of method 510 can be ranked according to probabilitiesassociated with each candidate word of the list of candidate words. Inan example, the probabilities reflect the likelihood that the candidateword is the word intended by the user.

At sub process 535 of method 510, the ranked list of candidate words canbe displayed to the user and a user selection of the intended word canbe made during word selection mode. In the case of a wearable device,the ranked list of candidate words can be displayed to the user via adisplay of the wearable device. The selection of the intended wordincludes tactile interaction by the user with the QWERTY keyboard of theTipText system and instruction regarding candidate words of the rankedlist of candidate words. Sub process 535 of method 510 will be describedin greater detail with respect to FIG. 5C.

The word selected at sub process 535 of method 510 can then be enteredinto a sentence being formed by the user at step 545 of method 510.

According to an embodiment, sub process 520 of method 510 will now bedescribed with reference to FIG. 5B. As noted in FIG. 5A, keyboard inputfrom the user can be received at step 515 of method 510. The keyboardinput can be a series of taps corresponding to an intended word. In theevent auto-complete is not engaged, or as a complement thereto, thekeyboard input may also be a specific tactile action indicating the userhas completed a present haptic input session. The received keyboardinput can then be evaluated by a statistical decoder, as introducedabove. The statistical decoder includes a spatial model, whichdescribes, as a probability, a relationship between touch locations of auser on the QWERTY keyboard and real locations of keys on the QWERTYkeyboard, and a language model, which provides probability distributionsof a sequence of words for a given language. In an example, the givenlanguage is the English language. Accordingly, the spatial model of thestatistical decoder may be applied to the received input at step 521 ofsub process 520 and the language model of the statistical decoder may beapplied to the received input, simultaneously, at step 522 of subprocess 520.

In an embodiment, and upon entry of a series of letters by a user, thestatistical decoder of sub process 520 of method 510 combinesprobabilities generated by each of the spatial model and the languagemodel at step 521 and step 522, respectively. The statistical decoderthen generates an overall probability of a word according to Bayes'theorem. In this way, the statistical decoder generates a list ofcandidate words, at step 523 of sub process 520, which can be ranked byoverall probability. As described with respect to step 530 of method510, a higher ranking of a candidate word indicates less ambiguityissues the input presented to the TipText system.

Moreover, and according to an embodiment of the present disclosure, thestatistical decoder finds a word W* in lexicon L for a given set oftouch points on the keyboard S=[1 . . . , sn . . . , sn] that satisfies:

$\begin{matrix}{W^{*} = {\arg\max\limits_{W \in L}{P( {W❘S} )}}} & (1)\end{matrix}$

From the Bayes' rule,

$\begin{matrix}{{P( {W{❘S}} )} = \frac{{P( {S{❘W}} )}{P(W)}}{P(S)}} & (2)\end{matrix}$

Since P(S) is an invariant across words, Equation (1) can be convertedto

$\begin{matrix}{W^{*} = {\arg\underset{W \in L}{\max}{P( {S❘W} )}{P(W)}}} & (3)\end{matrix}$where P(W) is obtained from a language model and P(S|W) is obtained froma spatial model, which can be calculated according to the followingapproach.

Assuming that W is comprised of n letters: c₁, c₂, c₃, . . . , c_(n), Shas n touch points, and each tap is independent, P(S|W) can be describedas

$\begin{matrix}{{P( {S{❘W}} )} = {{\prod}_{i = 1}^{n}{P( {s_{i}{❘c_{i}}} )}}} & (4)\end{matrix}$

It can be assumed that touch points for text entry using TipText followa similar pattern as text entry on a touchscreen. Therefore, if thecoordinates of s_(i) are (x_(i), y_(i)), P(s_(i)|c_(i)) can becalculated using a bivariate Gaussian distribution as

$\begin{matrix}{{P( {s_{i}{❘c_{i}}} )} = {\frac{1}{2\pi\sigma_{ix}\sigma_{iy}\sqrt{1 - \rho_{i}^{2}}}{\exp\lbrack {- \frac{z}{2( {1 - \rho_{i}^{2}} )}} \rbrack}}} & (5)\end{matrix}$where

$\begin{matrix}{z \equiv {\frac{( {x_{i} - \mu_{ix}} )^{2}}{\sigma_{ix}^{2}} - \frac{2{\rho_{i}( {x_{i} - \mu_{ix}} )}}{\sigma_{ix}\sigma_{iy}} + \frac{( {y_{i} - \mu_{iy}} )^{2}}{\sigma_{iy}^{2}}}} & (6)\end{matrix}$, (μ_(ix), μ_(iy)) is the center of the touch point distribution aimedon key and c_(i), σ_(ix) and σ_(iy) are standard deviations, and ρ_(i)is the correlation.

Separately, and as it relates to auto completion of words, the TipTextsystem assumes that users generate no insertion and omission errors andeach ‘key’ is tapped independently. Thus,

$\begin{matrix}{{P( {S{❘W}} )} = {{\prod}_{i = 1}^{n}{P( {S_{i}{❘W_{i}}} )} \times a^{m - n}}} & (8)\end{matrix}$where S_(i) refers to the ith letter of word entered by the user, andW_(i) refers to the ith letter of W in the dictionary with lengthbetween S and (S+8). The maximum length limit of 8 is arbitrary and maybe chosen according to testing. Finally, α refers to the penaltypreventing long words with high frequency of being ranked high, and m isthe length of W, wherein m≥n. α can be set to 0.7, thereby yielding thebest compromise between the aggressiveness and candidate coverage forthe TipText system.

In view of the above, and having generated the ranked list of candidatewords at step 530 of method 510, the user can be engaged during subprocess 535 of method 510.

At step 536 of sub process 535, the ranked list of candidate words canbe received. At step 537 of sub process 535, a first candidate word, orpresent candidate word, can be displayed to the user via a display ofthe TipText system. As described above, the display of the TipTextsystem may be on a wearable device such as smart glasses or a smartwatch. The user may indicate by haptic feedback, in the word selectionmode, whether the present candidate word is a word that was intended tobe conveyed. The haptic feedback regarding the present candidate wordcan be received at step 538 of sub process 535. If the received hapticfeedback indicates the present candidate word is the intended word, subprocess 535 continues to step 539 and the intended word, or selectedword, is saved in a buffer in preparation for appendage to an presentsentence structure. Alternatively, if the received haptic feedback atstep 538 of sub process 535 indicates the present candidate word is notthe intended word, sub process 535 of method 510 returns to step 537 anda subsequent candidate word can be displayed, as the present candidateword, to the user for evaluation. As the candidate words are in a rankedlist, the subsequent candidate word can be the next lower rankedcandidate word within the ranked list. Nevertheless, upon receivinghaptic input at step 538 of sub process 535 that the present candidateword is the intended word, sub process 535 of method 510 can proceed tostep 539 and the selected word can be saved in a buffer for appendage tothe present sentence structure.

According to an embodiment of the present disclosure, the haptic inputsreceived at step 538 of sub process 510 may be generated by differentactions of the user relative to the QWERTY keyboard. For instance, theuser may swipe a thumb across the QWERTY keyboard in order to indicate apresent haptic input session is complete and the word selection mode isready to be entered. In an example, the selection mode can be entered byswiping right across the QWERTY keyboard. In the word selection mode,the user can determine whether the present candidate word, or in thecase of the initial candidate word, the highest ranked candidate word,is the intended word. If the user determines the present candidate wordis the intended word, the user can proceed with the next word in asentence, tapping “keys” of the QWERTY keyboard that correspond to asubsequent intended word of the sentence. In an embodiment, a space willbe automatically inserted after the present candidate word becomes‘committed’ by user tapping of the subsequent intended word. If,alternatively, the user determines the present candidate word is not theintended word, the user can, again, swipe across the QWERTY keyboard inorder to summon the next candidate word from the ranked list ofcandidate words. In an example, the next candidate word can be summonedby swiping right across the QWERTY keyboard.

The real-word utility and applicability of the above-described TipTextsystem will now be demonstrated with reference to non-limitingexperimental design and experimental results.

Non-Limiting Experimental Results

According to an embodiment, two options were considered in designing ausable keyboard layout for the TipText system. A first option is todirectly adopt a layout with 26 keys. Although keys will be extremelyhard to select correctly, the intuition is that the statistical decodermay tolerate many, if not all, of the tapping errors, as shownpreviously for larger devices like smart watches and smartphones. Asecond option is to incorporate a larger size but smaller number of keysin a grid layout, similar to a T9 or a 1-line keyboard. The benefit ofthis option is that keys are larger, thus making it easier to acquire asignal. Ambiguity, however, may become an issue as each key isassociated with more than one letter. Each option was explored.

(i) Study One

Study One was conducted to explore the feasibility of directly adoptinga layout with 26 keys, in which data were collected reflecting eyes-freetyping behaviors on a miniature QWERTY keyboard. The goal thereof was tocollect data to understand eyes-free typing using the thumb-tip on akeyboard with 26 keys, thereby informing final keyboard design. Anothermotivation was to determine whether it is feasible for users to performtext entry based on their natural spatial awareness of a QWERTY layout,without practicing ahead of time on locations of keys. 10 right-handedparticipants (4 female) aged between 20 and 26 were recruited toparticipate in the study. The study was conducted with the Vicon motiontracking system for finger tracking with 1 mm accuracy and the Unity2018.3.5f1 game engine for real-time physical touch estimation.Interactive skin to sense user input, as described above with referenceto FIG. 3A through FIG. 3C, was intentionally not implemented in aneffort to minimize sensor influence on user spatial awareness of keylocations, as studies have found that user spatial acuity andsensitivity can be affected by the presence of the epidermal sensor.

With reference now to FIG. 6A, markers were attached on the nail of thethumb and index finger. As shown in FIG. 2B, markers 606 were attachedto the nail of a thumb 604 and to the nail of an index finger 603 inorder to obtain precise thumb-tip touch locations on the index finger603. A Vicon motion tracking system, including the tripods of FIG. 6A,was then able to track movements and orientation of the first segmentsof the thumb 604 and the index finger 603 of the user. Though it can beappreciated the motion tracking data can be acquired by another similarsystem, the data from Vicon was then used to control movement of eachfinger within a three-dimensional (3D) environment, wherein each fingerhas a 3D virtual representation. The virtual fingers werehigh-resolution 3D meshes of the index finger 603 and the thumb 604 ofthe user obtained by scanning, using a topographical scanner, claymodels of the index finger 603 and the thumb 604 of the user, as shownin FIG. 6C. In an example, a Roland Picza LPX-250RE laser scanner wasused as the topographical scanner.

Three meshes were used for real-time physical simulation during thestudy. It was observed that people used different thumb regions (e.g.thumb tip, side of the thumb) to perform touch input on the indexfinger. Accordingly, participants were allowed to tap using differentregions of the thumb to preserve a natural and comfortable interaction.With reference to FIG. 7 , when a thumb 704 was in contact with an indexfinger 703, a collision of the 3D finger meshes could be detected at atouch point 753. Ideally, the 3D meshes should deform accordingly toreflect the deformation, in practice, of the skin of the fingertips. Inthe simulation, such collisions were allowed to penetrate for the sakeof simplicity. The touch point 753 in a 3D space was estimated using thecenter of a contact area between the meshes calculated using a meshintersection algorithm. A touch event was registered based upon the sizeof the intersection exceeding a threshold value. The 3D touch point 753was then projected to a virtual plane perpendicular to the index fingersurface, representing a 2D keyboard. Since fingers of each participantwere different in size and shape, their fingers were manually measuredand a corresponding plane for each participant to fit the first segmentof the index finger 703 was transformed, accordingly. The projectionpoint, or touch point 753, captured using the local coordinate system ofthat plane, can be used as input of a participant. It can be appreciatedthat, while the estimation of tap location may not reflect the realsensor data from the interactive skin, it provided a reasonable estimateto inform the design of the keyboard layout.

During evaluation, eyes-free thumb-tip text entry tasks were performedwith four blocks of ten phrases using a Wizard of Oz keyboard (e.g., noreal keyboard was involved). The phrases were picked randomly fromMacKenzie's phrase set. The same set of 40 phrases was used for all theparticipants. For each letter, participants tapped on an imaginary keylocation on the first segment of the index finger using the thumb-tip oftheir dominant hand based on their natural spatial awareness. They wereasked to perform the task using their dominant hand as naturally aspossible and assume that the keyboard would correct input errors. Oursystem always displayed the correct letters no matter where they tapped.In a few cases, however, users accidentally touched the input area onthe finger before they were ready to input a new letter. Accordingly,the user was afforded a left swipe gesture to delete the last letter andallow users to correct these errors. After entering a phrase,participants pressed a “Done” button to proceed to the next phrase. Thisprocess was repeated until they completed all phrases. Participants wereencouraged to take a short break between blocks. During the study, amonitor was placed in front of the participant to display the task. Astatic image of a QWERTY keyboard was also shown on the monitor toremind participants about the positions of keys. Participants sat in achair with their dominant hand placed on the armrest and out of sight ofthe participant. Their finger could face any comfortable orientation. Anexperimenter sat beside them to ensure that their attention was on themonitor. Prior to the study, the system was calibrated for eachparticipant to ensure that the fingers and their virtual representationsin the 3D space were well aligned with each other. Before the study,participants were given a brief period of time to familiarize themselveswith the system without practicing locations of keys. Touch points wererecorded according to the local coordinates of 2D planes, which variedfrom user to user, and were normalized to obtain a general distribution.Touch points from ten participants, presented as scatter plots of touchpoints on a 26 key QWERTY keyboard, are shown in FIG. 8 . The touchlocations for different keys are shown in different hues. Thecorresponding letters are shown at centroids of the touch points alongwith a 95% confidence ellipse. It can be appreciated, that upon visualevaluation, touch locations are noisy with considerable overlaps amongdifferent ellipses, suggesting that eyes-free typing on a miniaturefingertip keyboard with 26 keys is imprecise. However, it is stillobservable that centroids of user touch points for 26 keys form a QWERTYlayout, except that some keys do not clearly separate apart from eachother. For example, “Y” and “U” almost overlap. A language model may behelpful in this case. The above results demonstrate that, though thekeys may be small, there is still a chance that participants might beable to type on a keyboard of 26 keys on the tip of the finger with thehelp of a statistical decoder. A general spatial model for this keyboardwas derived from the collected data and used, subsequently, to comparethe 26 key QWERTY keyboard with other approaches.

(i) Study Two

Study Two was conducted to explore the feasibility of incorporating alarger size keyboard but with fewer keys in a grid layout. In thislayout, keys are larger in size to facilitate tapping but fewer inquantity in order to fit into the same rectangular input space of theQWERTY keyboard. The results of this approach can be compared againstthe 26 key QWERTY keyboard of Study One. Note that larger keys mean thateach key may be associated with more than one letter. As such, userinput may become ambiguous as it is unclear which letter is the targetof the user. Therefore, a challenge of this approach is to find akeyboard layout that can best balance tapping precision and inputambiguities best.

There are 1,146,484 possible arrangements of a gridded rectangular spaceof a keyboard and assignments of 26 letters to each key per grid design.Accordingly, the theoretical performance of all possible arrangementswas considered, For each candidate keyboard design, a simulation firstcalculated the key entries per target word and then found a list ofwords that exactly matched the key entries due to input ambiguities. Ifthe list existed with more than one word, it was ordered by wordfrequency. No spatial information was involved at this step. The systemrecorded whether the target word appeared in the top three entries ofthe list. This approach repeated until it finished all the test wordspicked from a corpus. The test words may be, in an example, the top15,000 words from the American National Corpus, covering over 95% ofcommon English words. The percentage of times when the target wordappeared in the top three entries of the list was calculated as worddis-ambiguity scores for the given keyboard design.

As mentioned above, only the language model was used in the simulationtest since the spatial model of a statistical decoder cannot be acquiredwithout a user study. Accordingly, assuming that no tapping errorsexist, performance of the best candidate keyboard design is bounded byP(W), as the spatial mode P(S|W) is 1 in this scenario. Therefore, theassumption of this comparison test is that tapping errors do not existregardless of how small the keys are. In another embodiment, this can becorrected by incorporating heuristics, where top ranked candidates alsoneeded to have large keys.

After the simulator evaluated each of the possible keyboard designs thatconfined to the QWERTY's alphabetical arrangement, the arrangementswhich received a word disambiguation score higher than 90% wereselected. These designs included keyboards ranging from one row to threerows, among which, the ones with the least number of keys were selected,thereby striking a balance between key size and word disambiguity. Theremaining 162,972 candidates had a keyboard design in one of a 1×5 grid,a 2×3 grid, or a 3×2 grid. The keyboard layout of the top ranked design,which received a word disambiguity score of 94.6%, is shown in FIG. 43 .This score represents the theoretical upper bound of all possibledesigns in these three grids,

Note that an issue with this design is that many letters are shiftedaway from their original locations. For example, “G” and “V” are both inthe horizontal center of a QWERTY keyboard, but now neither of themresides inside the middle key in the second row. This is due to theresult of maximizing word disambiguity. The trade-off is learnability aspeople can no longer rely on their existing knowledge of the layout of aQWERTY keyboard. Instead, new letter locations will have to be learnedupon initiating eyes-free typing. An extra design criterion was thusconsidered, which restricted letter assignments to follow their originallocations strictly unless the letter resides at the boundary of two keys(e.g., “G” originally resides on the boundary of the two keys in thesecond row under a 3×2 grid). In this case, the possibilities for theletter to be assigned to either key were considered. By applying thisrule, only 50 qualified out of all 162,972 candidates. This included 16for the 1×5 grid, 32 for the 2×3 grid, and 2 for the 3×2 grid.

Subsequently, an understanding of user natural spatial awareness of keylocations in these three grid layouts was obtained. Specifically, theunderstanding included knowing how grids differ with regard to tappingprecision. The answer to these questions aided in the derivation of aspatial model for each of the three candidate grid layouts, which couldbe used to form a complete statistical decoder with the language modelto estimate the performance of the different keyboard designs associatedwith these grids.

Accordingly, as a goal of Study Two, a spatial model for each of thesethree grid layouts was derived. Initially, the assignment of 26 lettersto grid keys was yet to be determined and, therefore, the text entrytask was replaced by a target acquisition task in which participantswere instructed to acquire cells in a grid. As such, the spatial modelsobtained from Study Two served as a close approximation of the spatialmodels for acquiring keyboard keys, which were identical in size andlocation as the grid cells. 12 right-handed participants (4 female) agedfrom 20 to 26, for counterbalancing grid conditions, were recruited. Theapparatus of Study One (i.e., FIG. 6A through FIG. 6C) was used in StudyTwo.

During Study Two, participants were required to select a target cell inone of the three tested grid layouts by tapping somewhere on the firstsegment of the index finger using the thumb-tip of their dominant hand.Because letter assignment was not considered, targets were generated ina random order instead of following a corpus. The grid layouts wereintroduced to participants by an experimenter describing the number ofrows and columns. During the study, no visual grid layout was shown tothe user. Instead the target was indicated by row and column number toavoid influencing tapping behaviors of each participant. Participantswere asked to perform the task using their dominant hand as fast and asaccurately as possible without looking at their fingers. Upon the end ofa trial, a new target appeared. This process was repeated untilparticipants completed all trials. Prior to the study, participants weregiven a brief period of time (e.g. 5 to 10 minutes) to familiarizethemselves with the system and the representation of location in row andcolumn number.

Study Two employed three grid layout conditions: 1×5, 2×3, and 3×2. Theorder of the three conditions was counter-balanced among participantsand the target location was presented randomly. Each target in a gridrepeated 50 times. FIG. 9A through FIG. 9C show the distributions of alltouch points from 12 participants for each of three grid layouts. Thetouch locations for different cells are shown in different hues. Thecentroids of points for all cells and the 95% confidence ellipses arealso shown. As expected, the touch locations are less noisy than on theQWERTY layout with 26 keys (i.e., Study One). There is, however, stilloverlap among the ellipses for all three grids. This suggests thattapping on the tested grid layouts based on participant imagination andspatial awareness is still inaccurate, to an extent. Separation of thetouch points, however, is improved. Among the three tested grids, lessoverlap was observed on the 2×3 and 3×2 grids than on the 1×5 grid,possibly due to the cells of these grids being wider. It can be notedthat centroids of touch points are all well separated for all threegrids, following the same geometry of the three tested grids andsuggesting that participants were able to identify the location of thegrid cells using spatial awareness without looking at their fingers.Although tapping precision tended to be low, a keyboard decoder would beexpected to tolerate the errors. A general spatial model was derived foreach grid layout using the data collected in Study Two. The spatialmodel and the language model were used to form a statistical decoder,which was used subsequently to identify the most suitable keyboarddesign for the TipText system. 100981 With the general statisticaldecoders obtained for the keyboard with 26 keys (default keyboard) andthe three grid layouts, another simulation was conducted, in which textentry on the default keyboard by the 10 participants from Study One andon the 50 grid candidates by the 12 participants from Study Two wassimulated. It was assumed that typing using the TipText system issimilar to typing on a soft keyboard in that user touch locations followa bivariate Gaussian distribution. Therefore, the location of the usertouch input was generated based on the bivariate Gaussian distributionof the individual spatial model. For each target word, the generatedtouch points served as input for the statistical decoder and thesimulation checked whether the target word appeared in the top threeentries of the list. The process was repeated similarly to the firstsimulation. As the touch points generated from different participantmodels are different, the word disambiguity scores for each candidatekeyboard layout differed among participants. Therefore, an average scorewas calculated to represent the performance of each candidate. Thedefault keyboard received an average score of 71.6%. On the other hand,among the 50 grid layout candidates, 10 grid layouts had a disambiguityscore above 80%. All grid layouts were in a 2×3 grid. The top rankedlayout, shown in FIG. 10A, scored an average of 82.38%. It was also theone scored the highest by 9 out of 12 participants. The winning layoutoutperformed the one ranked the lowest, shown in FIG. 10B, by 45.83%, Italso outperformed the default layout by 10.78%. Therefore, the gridlayout of FIG. 10A was used for the TipText system.

(iii) Study Three

In view of the above selected grid layout of FIG. 10A, an interactiveskin overlay for the TipText system was developed. In an exemplaryembodiment, the thin and flexible device measures ˜2.2 cm×2.2 cm andcontains a printed 3×3 capacitive touch sensor matrix. The sensorfeatures diamond shaped electrodes of 5 mm diameter and 6.5 mmcenter-to-center spacing. The interactive skin overlay featured aflexible printed circuit, as shown in FIG. 11 , thereby providing morereliable readings on sensor data. Such flexible printed circuit measured0.025-0.125 mm thick and 21.5 mm×27 mm wide.

The finished sensor was controlled using an Arduino Nano with a MPR121touch sensing chip. The raw capacitive data from each channel wastransmitted at a frequency of 100 Hz. Software that interpolates theelectrode data was implemented in C#. Of course, it can be appreciatedthat software and hardware of the device of FIG. 13 would be similarlycapable of performing the methods described herein.

In evaluating the TipText system implementing the flexible circuit boardof FIG. 11 , a user study was conducted to evaluate performance. Suchstudy also allowed for measurement of how well the selected keyboarddesign worked on a state-of-the-art micro thumb-tip gesture sensor. 12right-handed participants (2 female) aged between 20 and 27 wererecruited, each participant being previously familiar with the QWERTYkeyboard. The study was conducted using the interactive skin prototypeof FIG. 11 , developed using flexible printed circuit. During the study,participants sat in a chair and placed their hands similarly to that ofStudy One described herein. An experimenter sat beside the participantto ensure that the attention of the participant was on the monitor. Testphrases and top three candidates were shown on a monitor placed at acomfortable distance from the participant which simulated the situationwhere a near-eye display is available to the user. Swipe gestures wereused to allow participants to navigate the candidate list and delete thelast entered letter. A static image of the keyboard was shown on themonitor to remind participants about the positions of keys duringtraining while it was hidden during the study.

The touch sensor of the interactive skin of the TipText system wascalibrated for each participant prior to the study by having them tapthree edge locations on the first segment of the index finger (e.g., tipand the two ends of the edge of the segment). This was to ensure thatthe sensor readings of the interactive skin were largely aligned withthe spatial model previously obtained. Prior to the experiment,participants were allowed to practice at their discretion. During thestudy, participants transcribed 4 blocks, each containing 10 phrasespicked randomly from MacKenzie's phrase set. The same set of 40 phraseswas used for all participants. No phrase was repeated. After entering aphrase, participants pressed the button of a mouse placed on a tablewith their non-wearing hands to proceed to the next phrase. This processwas repeated until they completed all the phrases. The experimentalsession lasted approximately 40 minutes, depending on participant speed.480 phrases (12 participants×4 blocks×10 phrases) were collected in thestudy.

The resulting data were analyzed using one-way repeated measures ANOVAand Bonferroni corrections for pair-wise comparisons. For violations tosphericity, we used a Greenhouse-Geisser adjustment for degrees offreedom. ANOVA yielded a significant effect of Block (F(3)=20.529,p<0.001). The average text entry speed was 11.9 WPM (standard error of0.5). FIG. 12A is a graphical illustration of the mean WPM by block,demonstrating a performance improvement with practice. For instance,post-hoc pair-wise comparisons showed a significant difference betweenfirst and second block (p<0.05). To this end, participants achieved 10.5WPM (standard error of 0.6) in the first block and the speed increasedto 13.3 WPM (standard error of 0.5) in the last block with animprovement of 27%. Accordingly, it can be said that participants wereable to achieve a fairly good speed even in the first block, suggestingthat participants were able to pick up text entry via the TipText systemrelatively quickly. Error rate is reported based on uncorrected errorrate (UER) and total error rate (TER), shown in FIG. 12B. Uncorrectederrors were the errors found in the final input phrases whereas totalerrors included both corrected and uncorrected errors. ANOVA yielded asignificance effect of Block on TER (F(3)=4.986, p<0.01). Typing speedincreased with the decrease in errors. This suggests that correctingerrors was the major source that prevented participants from typingfaster, though participants were generally able to identify errors andcorrect them. This is based on no significant effect of Block on UER(F(3)=2.396, p>0.05). Overall, the average TER and UER was 4.89%(standard error of 0.66%) and 0.30% (standard error of 0.33%)respectively. Noted above, FIG. 12B shows TER and UER by block. Theaverage TER in the first block was 6.75% (standard error of and itimproved significantly in the last block (3.88%, standard error of0.53%). The average UER was 0.30% (standard error of 0.33%), which didnot change significantly across all blocks. This suggests that when atarget word fell outside of the top three suggestions, participantstended to delete the word and retype instead of exploring further downthe list even if the candidate was sometimes only a swipe away. Thisprovides some insight as to the optimal number of candidate words thatshould be shown.

Auto-complete rate of a word was calculated by dividing the number ofautomatically filled letters by the length of that word. The overallauto-complete rate was thus the mean of the auto-complete rate of alltested words. Overall, the auto-complete rate was 14.91% (standard errorof 2.39%) for all the input words across all four blocks. We found thattext entry speed without auto-complete on Block 4 was13.3×(100%−14.91%)=11.3 WPM. There was no significant effect of Block onauto-complete (F(3)=2.406, p>0.05). Over the four blocks, the meanstandard deviation was 0.74%. This suggested that participants usedauto-complete consistently throughout even getting more familiar withthe keyboard layout.

With regard to text entry speed and error rate, the average speed oftext entry via the TipText system was 11.9 WPM, though participants wereable to achieve 13.3 WPM in the last block. This is faster than theexisting finger-based one-handed text-entry technique, FingerT9 (5.42WPM), which uses the entire body of all four fingers as the input spacefor a keypad. The performance of text entry via the TipText system isalso comparable with DigiTouch, a bimanual text entry technique usingthe fingers of both hands (average 13 WPM). In the context of mobilescenarios, the TipText system has the advantage of freeing the otherhand of the user for other tasks, such as carrying shopping bags. Asparticipants were able to pick up text entry via the TipText systemquickly and without seeing a keyboard, the TipText system might be agood option for ultra-small devices without a screen. Further, theseresults show an improving trend for speed, suggesting that expertperformance could be even higher and, thus, warrant a longer-term study.Accordingly, future research should consider the upper boundary of textentry input speed via the TipText system.

With regard to the number of suggestions, and considering that thenumber of suggestions could affect the layout performance becausesearching through the candidate word list requires extra cognitiveeffort and visual attention, the present disclosure consideredpresentation of three candidate words. However, since the TipText systemwas designed to avoid showing an on-screen keyboard on a small computingdevice (e.g., a smart watch or smart glasses), it is thus possible thatmore than three candidate words can be shown to the user. Furtherresearch should consider how the number of suggestions may affect typingperformance and whether an optimal number of suggestions exist for ageneral population.

With regard to the statistical decoder, the present disclosure describesa statistical decoder derived from the general spatial data collectedfrom twelve participants. The bivariate Gaussian distributions varyamong different users and a personalized keyboard decoder cantheoretically improve typing performance for each individual. In anembodiment, an adaptive algorithm that can effectively shift the modelfrom general to personal may be developed. Additionally, it may beimportant to further investigate adaptive algorithms that candynamically update the statistical decoder according to instantaneousand historical input from each user, as tapping behaviors of each usermay vary with different hand postures and contexts such as standing andwalking.

According to an embodiment, the present disclosure describes a microthumb-tip text entry technique based on a miniature invisible keyboardresiding invisibly on the first segment of the index finger. Theminiature invisible keyboard optimizes layout learnability, key size,and word dis-ambiguity, and includes a 23 grid layout with the lettershighly confining to the alphabetic and spatial arrangement of QWERTY.The design of this keyboard was optimized for eves-free input byutilizing a spatial model reflecting users' natural spatial awareness ofkey locations on the index finger so the user does not need to look atthe keyboard when typing. It is anticipated that micro finger gesturetyping has many applications, ranging from mobile, wearable, and AR.

Next, a hardware description of an apparatus of the TipText system,according to exemplary embodiments, is described with reference to FIG.13 . In FIG. 13 , the apparatus of the TipText system may be a wearabledevice, such as a smart watch or smart glasses, and the computationalresources of the wearable device, in communication with an interactiveskin of the TipText system arranged on the finger of the user, may beexploited in order to perform the methods of the TipText systemdescribed above. To this end, the apparatus of the TipText systemincludes a CPU 1380 which performs the processes described above/below.The process data and instructions may be stored in memory 1381. Theseprocesses and instructions may also be stored on a storage medium disk1382 such as a hard drive (HDD) or portable storage medium or may bestored remotely. Further, the claimed advancements are not limited bythe form of the computer-readable media on which the instructions of theinventive process are stored. For example, the instructions may bestored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM,hard disk or any other information processing device with which theapparatus of the TipText system communicates, such as a server orcomputer.

Further, the claimed advancements may be provided as a utilityapplication, background daemon, or component of an operating system, orcombination thereof, executing in conjunction with CPU 1380 and anoperating system such as Microsoft Windows 7, UNIX, Solaris, LINUX,Apple MAC-OS and other systems known to those skilled in the art.

The hardware elements in order to achieve the apparatus of the TipTextsystem may be realized by various circuitry elements, known to thoseskilled in the art. For example, CPU 1380 may be a Xenon or Coreprocessor from Intel of America or an Opteron processor from AMD ofAmerica, or may be other processor types that would be recognized by oneof ordinary skill in the art. Alternatively, the CPU 1380 may beimplemented on an FPGA, ASIC, PLD or using discrete logic circuits, asone of ordinary skill in the art would recognize, Further, CPU 1380 maybe implemented as multiple processors cooperatively working in parallelto perform the instructions of the inventive processes described above.

The apparatus of the TipText system in FIG. 13 also includes a networkcontroller 1383, such as an Intel Ethernet PRO network interface cardfrom Intel Corporation of America, for interfacing with network 1395. Ascan be appreciated, the network 1395 can be a public network, such asthe Internet, or a private network such as an LAN or WAN network, or anycombination thereof and can also include PSTN or ISDN sub-networks. Thenetwork 1395 can also be wired, such as an Ethernet network, or can bewireless such as a cellular network including EDGE, 3G and 4G wirelesscellular systems. The wireless network can also be WiFi, Bluetooth, orany other wireless form of communication that is known.

The apparatus of the TipText system further includes a displaycontroller 1384, such as a NVIDIA GeForce GTX or Quadro graphics adaptorfrom NVIDIA Corporation of America for interfacing with display 1384,such as a Hewlett Packard HPL2445w CD monitor. In an embodiment, thedisplay 1384 may be a display of the wearable device, such as a smartwatch or smart glasses, and may be used for displaying the ranked listof candidate words. A general purpose I/O interface 1386 interfaces witha keyboard 1387 as well as a touch screen panel 1388 on or separate fromdisplay 1385. In an embodiment, the keyboard 1387 may be a QWERTYkeyboard integrated within an interactive skin of the TipText system.The keyboard 1387 may be in wired or wireless communication with thegeneral purpose I/O interface 1386. General purpose I/O interface 1386also connects to a variety of peripherals 1389. The peripherals 1389 mayinclude certain other aspects of the interactive skin of the TipTextsystem of the present disclosure, independently or in combination withthe above-described features. For instance, the peripherals 1389 mayinclude supplemental controls to enable wired or wireless interactionwith the tactile sensors of the QWERTY keyboard.

A sound controller 1390 is also provided in the apparatus of the TipTextsystem, such as Sound. Blaster X-Fi Titanium from Creative, to interfacewith speakers/microphone 1391 thereby providing sounds and/or music.

The general purpose storage controller 1392 connects the storage mediumdisk 1382 with communication bus 1393, which may be an ISA, EISA, VESA,PCI, or similar, for interconnecting all of the components of theapparatus of the TipText system. A description of the general featuresand functionality of the display 1385, keyboard 1387, as well as thedisplay controller 1384, storage controller 1392, network controller1383, sound controller 1390, and general purpose I/O interface 1386 isomitted herein for brevity as these features are known.

According to an embodiment, the apparatus of the present disclosure is asmart device, such as smart glasses or a smart watch. FIG. 14 is a moredetailed block diagram illustrating a smart device 20, an exemplary userdevice 20, according to certain embodiments of the present disclosure.In certain embodiments, user device 20 may be a smartphone. However, theskilled artisan will appreciate that the features described herein maybe adapted to be implemented on other devices (e.g., a laptop, a tablet,a server, an e-reader, a camera, a navigation device, a smart watch, asmart glasses, etc.). The exemplary user device 20 of FIG. 14 includes acontroller 110 and a wireless communication processor 102 connected toan antenna 101. A speaker 104 and a microphone 105 are connected to avoice processor 103.

The controller 110 is an example of the control unit 21 and may includeone or more Central Processing Units (CPUs), and may control eachelement in the user device 20 to perform functions related tocommunication control, audio signal processing, control for the audiosignal processing, still and moving image processing and control, andother kinds of signal processing. The controller 110 may perform thesefunctions by executing instructions stored in a memory 150.Alternatively or in addition to the local storage of the memory 150, thefunctions may be executed using instructions stored on an externaldevice accessed on a network or on a non-transitory computer readablemedium.

The memory 150 includes but is not limited to Read Only Memory (ROM),Random Access Memory (RAM), or a memory array including a combination ofvolatile and non-volatile memory units. The memory 150 may be utilizedas working memory by the controller 110 while executing the processesand algorithms of the present disclosure. Additionally, the memory 150may be used for long-term storage, e.g., of image data and informationrelated thereto.

The user device 20 includes a control line CL and data line DL asinternal communication bus lines. Control data to/from the controller110 may be transmitted through the control line CL. The data line DL maybe used for transmission of voice data, display data, etc.

The antenna 101 transmits/receives electromagnetic wave signals betweenbase stations for performing radio-based communication, such as thevarious forms of cellular telephone communication. The wirelesscommunication processor 102 controls the communication performed betweenthe user device 20 and other external devices via the antenna 101. Forexample, the wireless communication processor 102 may controlcommunication between base stations for cellular phone communication.

The speaker 104 emits an audio signal corresponding to audio datasupplied from the voice processor 103. The microphone 105 detectssurrounding audio and converts the detected audio into an audio signal.The audio signal may then be output to the voice processor 103 forfurther processing. The voice processor 103 demodulates and/or decodesthe audio data read from the memory 150 or audio data received by thewireless communication processor 102 and/or a short-distance wirelesscommunication processor 107. Additionally, the voice processor 103 maydecode audio signals obtained by the microphone 105.

The exemplary user device 20 may also include a display 120, a touchpanel 130, an operation key 140, and a short-distance communicationprocessor 107 connected to an antenna 106. The display 120 may be aLiquid Crystal Display (LCD), an organic electroluminescence displaypanel, or another display screen technology. In addition to displayingstill and moving image data, the display 120 may display operationalinputs, such as numbers or icons which may be used for control of theuser device 20. The display 120 may additionally display a GUI for auser to control aspects of the user device 20 and/or other devices.Further, the display 120 may display characters and images received bythe user device 20 in response to haptic inputs of a user via aninteractive skin of the TipText system and/or stored in the memory 150(i.e. buffer of selected words) or accessed from an external device on anetwork. For example, the user device 20 may access a network such asthe Internet and display text and/or images transmitted from a Webserver.

The touch panel 130 may include a physical touch panel display screenand a touch panel driver. The touch panel 130 may include one or moretouch sensors for detecting an input operation on an operation surfaceof the touch panel display screen. The touch panel 130 also detects atouch shape and a touch area. Used herein, the phrase “touch operation”refers to an input operation performed by touching an operation surfaceof the touch panel display with an instruction object, such as a finger,thumb, or stylus-type instrument. In the case where a stylus or the likeis used in a touch operation, the stylus may include a conductivematerial at least at the tip of the stylus such that the sensorsincluded in the touch panel 130 may detect when the stylusapproaches/contacts the operation surface of the touch panel display(similar to the case in which a finger is used for the touch operation).

In certain aspects of the present disclosure, the touch panel 130 may bedisposed adjacent to the display 120 (e.g., laminated) or may be formedintegrally with the display 120. For simplicity, the present disclosureassumes the touch panel 130 is formed integrally with the display 120and therefore, examples discussed herein may describe touch operationsbeing performed on the surface of the display 120 rather than the touchpanel 130. However, the skilled artisan will appreciate that this is notlimiting.

For simplicity, the present disclosure assumes the touch panel 130 is acapacitance-type touch panel technology. However, it should beappreciated that aspects of the present disclosure may easily be appliedto other touch panel types (e.g., resistance-type touch panels) withalternate structures. In certain aspects of the present disclosure, thetouch panel 130 may include transparent electrode touch sensors arrangedin the X-Y direction on the surface of transparent sensor glass.

The touch panel driver may be included in the touch panel 130 forcontrol processing related to the touch panel 130, such as scanningcontrol. For example, the touch panel driver may scan each sensor in anelectrostatic capacitance transparent electrode pattern in theX-direction and Y-direction and detect the electrostatic capacitancevalue of each sensor to determine when a touch operation is performed.The touch panel driver may output a coordinate and correspondingelectrostatic capacitance value for each sensor. The touch panel drivermay also output a sensor identifier that may be mapped to a coordinateon the touch panel display screen. Additionally, the touch panel driverand touch panel sensors may detect when an instruction object, such as afinger is within a predetermined distance from an operation surface ofthe touch panel display screen. That is, the instruction object does notnecessarily need to directly contact the operation surface of the touchpanel display screen for touch sensors to detect the instruction objectand perform processing described herein. For example, in certainembodiments, the touch panel 130 may detect a position of a user'sfinger around an edge of the display panel 120 (e.g., gripping aprotective case that surrounds the display/touch panel). Signals may betransmitted by the touch panel driver, e.g. in response to a detectionof a touch operation, in response to a query from another element basedon timed data exchange, etc.

The touch panel 130 and the display 120 may be surrounded by aprotective casing, which may also enclose the other elements included inthe user device 20. In certain embodiments, a position of the user'sfingers on the protective casing (but not directly on the surface of thedisplay 120) may be detected by the touch panel 130 sensors.Accordingly, the controller 110 may perform display control processingdescribed herein based on the detected position of the user's fingersgripping the casing. For example, an element in an interface may bemoved to a new location within the interface e.g., closer to one or moreof the fingers) based on the detected finger position.

Further, in certain embodiments, the controller 110 may be configured todetect which hand is holding the user device 20, based on the detectedfinger position. For example, the touch panel 130 sensors may detect aplurality of fingers on the left side of the user device (e.g., on anedge of the display 120 or on the protective casing), and detect asingle finger on the right side of the user device 20. In this exemplaryscenario, the controller 110 may determine that the user is holding theuser device 20 with his/her right hand because the detected grip patterncorresponds to an expected pattern when the user device 20 is held onlywith the right hand.

The operation key 140 may include one or more buttons or similarexternal control elements, which may generate an operation signal basedon a detected input by the user. In addition to outputs from the touchpanel 130, these operation signals may be supplied to the controller 110for performing related processing and control. In certain aspects of thepresent disclosure, the processing and/or functions associated withexternal buttons and the like may be performed by the controller 110 inresponse to an input operation on the touch panel 130 display screenrather than the external button, key, etc. In this way, external buttonson the user device 20 may be eliminated in lieu of performing inputs viatouch operations, thereby improving water-tightness.

The antenna 106 may transmit/receive electromagnetic wave signalsto/from other external apparatuses, and the short-distance wirelesscommunication processor 107 may control the wireless communicationperformed between the other external apparatuses. Bluetooth, IEEE802.11, and near-field communication (NFC) are non-limiting examples ofwireless communication protocols that may be used for inter-devicecommunication via the short-distance wireless communication processor107.

The user device 20 may include a motion sensor 108. The motion sensor108 may detect features of motion (i.e., one or more movements) of theuser device 20. For example, the motion sensor 108 may include anaccelerometer to detect acceleration, a gyroscope to detect angularvelocity, a geomagnetic sensor to detect direction, a geo-locationsensor to detect location, etc., or a combination thereof to detectmotion of the user device 20. In certain embodiments, the motion sensor108 may generate a detection signal that includes data representing thedetected motion. For example, the motion sensor 108 may determine anumber of distinct movements in a motion (e.g., from start of the seriesof movements to the stop, within a predetermined time interval, etc.), anumber of physical shocks on the user device 20 (e.g., a jarring,hitting, etc., of the electronic device), a speed and/or acceleration ofthe motion (instantaneous and/or temporal), or other motion features.The detected motion features may be included in the generated detectionsignal. The detection signal may be transmitted, e.g., to the controller110, whereby further processing may be performed based on data includedin the detection signal. The motion sensor 108 can work in conjunctionwith a Global Positioning System (GPS) section 160. The GPS section 160detects the present position of the terminal device 100. The informationof the present position detected by the GPS section 160 is transmittedto the controller 110. An antenna 161 is connected to the GPS section160 for receiving and transmitting signals to and from a GPS satellite.

The user device 20 may include a camera section 109, which includes alens and shutter for capturing photographs of the surroundings aroundthe user device 20. In an embodiment, the camera section 109 capturessurroundings of an opposite side of the user device 20 from the user.The images of the captured photographs can be displayed on the displaypanel 120. A memory section saves the captured photographs. The memorysection may reside within the camera section 109 or it may be part ofthe memory 150. The camera section 109 can be a separate featureattached to the user device 20 or it can be a built-in camera feature.

The user device 20 may include a haptic section 170, comprisingprocessing circuitry and a tactile sensor and controller for detectingand receiving tactile interactions between a user and an interactiveskin of the TypText system. In an embodiment, the haptic section 170receives haptic input front the user and transmits data corresponding tothe haptic input to other processors for evaluation and candidate wordgeneration. In an embodiment, the haptic section 170 receives hapticinput from the user and performs evaluation and candidate wordgeneration locally.

Obviously, numerous modifications and variations are possible in lightof the above teachings. It is therefore to be understood that within thescope of the appended claims, the invention may be practiced otherwisethan as specifically described herein.

Embodiments of the present disclosure may also be as set forth in thefollowing parentheticals.

(1) A system for eyes-free text entry, comprising a wearable devicehaving a display, and processing circuitry configured to receive ahaptic input provided to a keyboard mounted on a finger of a user, thehaptic input being an indication of an alphabetical letter determinedbased on a location of the haptic input on the keyboard, generate a listof candidate words based on the received haptic input, each candidateword of the list of candidate words being associated with a probabilitythereof, display the generated list of candidate words to the user viathe display of the wearable device, receive a selection of a particularcandidate word of the list of candidate words, and append the particularcandidate word of the list of candidate words corresponding to thereceived selection to a present sentence structure, wherein the keyboardhas a layout based on a spatial model reflecting spatial awareness, bythe user, of key locations on the finger.

(2) The system according to (1), wherein the processing circuitry isconfigured to calculate the probability associated with each candidateword of the list of candidate words by generating a probability based onan application of the spatial model to the received haptic input, thespatial model describing a relationship between touch locations of theuser and locations of keys of the keyboard, generating a probabilitybased on an application of a language model to the received hapticinput, the language model providing probability distributions of asequence of words for a given language, and combining the generatedprobability based on the application of the spatial model and thegenerated probability based on the application of the language model togenerate the probability associated with each candidate word of the listof candidate words.

(3) The system according to either (1) or (2), wherein the processingcircuitry is further configured to rank each candidate word of thegenerated list of candidate words based on a respective probability ofeach candidate word.

(4) The system according to any one of (1) to (3), wherein theprocessing circuitry is further configured to receive a correctivehaptic input to the keyboard indicating that a prior haptic input shouldbe ignored, the corrective haptic input being a swipe of a thumb of theuser.

(5) The system according to any one of (1) to (4), wherein theprocessing circuitry is further configured to receive a directive hapticinput to the keyboard indicating that a candidate word of the generatedlist of candidate words is incorrect, the directive haptic input being aswipe of a thumb of the user.

(6) The system according to any one of (1) to (5), wherein the layout ofthe keyboard is a 2×3 grid and is based on QWERTY.

(7) The system according to any one of (1) to (6), wherein the keyboardis disposed within a flexible printed circuit.

(8) A method of eyes-free text entry, comprising receiving, byprocessing circuitry, a haptic input provided to a keyboard mounted on afinger of a user, the haptic input being an indication of analphabetical letter determined based on a location of the haptic inputon the keyboard, generating, by the processing circuitry, a list ofcandidate words based on the received haptic input, each candidate wordof the list of candidate words being associated with a probabilitythereof, displaying, by the processing circuitry, the generated list ofcandidate words to the user via a display of a wearable device,receiving, by the processing circuitry, a selection of a particularcandidate word of the list of candidate words, and appending, by theprocessing circuitry, the particular candidate word of the list ofcandidate words corresponding to the received selection to a presentsentence structure, wherein the keyboard has a layout based on a spatialmodel reflecting spatial awareness, by the user, of key locations on thefinger.

(9) The method according to (8), further comprising receiving, by theprocessing circuitry, a corrective haptic input to the keyboardindicating that a prior haptic input should be ignored, the correctivehaptic input being a swipe of a thumb of the user.

(10) The method according to either (8) or (9), further comprisingreceiving, by the processing circuitry, a directive haptic input to thekeyboard indicating that a candidate word of the generated list ofcandidate words is incorrect, the directive haptic input being a swipeof a thumb of the user.

(11) The method according to any one of (8) to (10), further comprisingranking, by the processing circuitry, each candidate word of thegenerated list of candidate words based on a respective probability ofeach candidate word.

(12) The method according to any one of (8) to (11), further comprisingdisplaying, by the processing circuitry, a ranked candidate word of thegenerated list of candidate words to the user via the display of thewearable device.

(13) The method according to any one of (8) to (12), wherein theprobability associated with each candidate word of the generated list ofcandidate words is calculated by generating, by the processingcircuitry, a probability based on an application of the spatial model tothe received haptic input, the spatial model describing a relationshipbetween touch locations of the user and locations of keys of thekeyboard, generating, by the processing circuitry, a probability basedon an application of a language model to the received haptic input, thelanguage model providing probability distributions of a sequence ofwords for a given language, and combining, by the processing circuitry,the generated probability based on the application of the spatial modeland the generated probability based on the application of the languagemodel to generate the probability associated with each candidate word ofthe list of candidate words.

(14) An apparatus for eyes-free text entry, comprising processingcircuitry configured to receive a haptic input provided to a keyboardmounted on a finger of a user, the haptic input being an indication ofan alphabetical letter determined based on a location of the hapticinput on the keyboard, generate a list of candidate words based on thereceived haptic input, each candidate word of the list of candidatewords being associated with a probability thereof, display the generatedlist of candidate words to the user via a display of a wearable device,receive a selection of a particular candidate word of the list ofcandidate words, and append the particular candidate word of the list ofcandidate words corresponding to the received selection to a presentsentence structure, wherein the keyboard has a layout based on a spatialmodel reflecting spatial awareness, by the user, of key locations on thefinger.

(15) The apparatus according to (14), wherein the processing circuitryis configured to calculate the probability associated with eachcandidate word of the list of candidate words by generating aprobability based on an application of the spatial model to the receivedhaptic input, the spatial model describing a relationship between touchlocations of the user and locations of keys of the keyboard, generatinga probability based on an application of a language model to thereceived haptic input, the language model providing probabilitydistributions of a sequence of words for a given language, and combiningthe generated probability based on the application of the spatial modeland the generated probability based on the application of the languagemodel to generate the probability associated with each candidate word ofthe list of candidate words.

(16) The apparatus according to either (14) or (15), wherein theprocessing circuitry is further configured to rank each candidate wordof the generated list of candidate words based on a respectiveprobability of each candidate word.

(17) The apparatus according to any one of (14) to (16), wherein theprocessing circuitry is further configured to receive a correctivehaptic input to the keyboard indicating that a prior haptic input shouldbe ignored, the corrective haptic input being a swipe of a thumb of theuser.

(18) The apparatus according to any one of (14) to (17), wherein theprocessing circuitry is further configured to receive a directive hapticinput to the keyboard indicating that a candidate word of the generatedlist of candidate words is incorrect, the directive haptic input being aswipe of a thumb of the user.

(19) The apparatus according to any one of (14) to (18), wherein thelayout of the keyboard is a 2×3 grid and is based on QWERTY.

(20) The apparatus according to any one of (14) to (19), wherein thekeyboard is disposed within a flexible printed circuit.

Thus, the foregoing discussion discloses and describes merely exemplaryembodiments of the present invention. As will be understood by thoseskilled in the art, the present invention may be embodied in otherspecific forms without departing from the spirit or essentialcharacteristics thereof. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting of the scopeof the invention, as well as other claims. The disclosure, including anyreadily discernible variants of the teachings herein, defines, in part,the scope of the foregoing claim terminology such that no inventivesubject matter is dedicated to the public.

The invention claimed is:
 1. A system for eyes-free text entry,comprising: a wearable device having a display; and processing circuitryconfigured to receive a haptic input provided to a keyboard mounted on afinger of a user, the haptic input being an indication of analphabetical letter determined based on a location of the haptic inputon the keyboard, generate a list of candidate words based on thereceived haptic input, each candidate word of the list of candidatewords being associated with a probability thereof, display the generatedlist of candidate words to the user via the display of the wearabledevice, receive a selection of a particular candidate word of the listof candidate words, and append the particular candidate word of the listof candidate words corresponding to the received selection to a presentsentence structure, wherein the keyboard has a layout based on a spatialmodel reflecting spatial awareness, by the user, of key locations on thefinger.
 2. The system according to claim 1, wherein the processingcircuitry is configured to calculate the probability associated witheach candidate word of the list of candidate words by generating aprobability based on an application of the spatial model to the receivedhaptic input, the spatial model describing a relationship between touchlocations of the user and locations of keys of the keyboard, generatinga probability based on an application of a language model to thereceived haptic input, the language model providing probabilitydistributions of a sequence of words for a given language, and combiningthe generated probability based on the application of the spatial modeland the generated probability based on the application of the languagemodel to generate the probability associated with each candidate word ofthe list of candidate words.
 3. The system according to claim I, whereinthe processing circuitry is further configured to rank each candidateword of the generated list of candidate words based on a respectiveprobability of each candidate word.
 4. The system according to claim I,wherein the processing circuitry is further configured to receive acorrective haptic input to the keyboard indicating that a prior hapticinput should be ignored, the corrective haptic input being a swipe of athumb of the user.
 5. The system according to claim 1, wherein theprocessing circuitry is further configured to receive a directive hapticinput to the keyboard indicating that a candidate word of the generatedlist of candidate words is incorrect, the directive haptic input being aswipe of a thumb of the user.
 6. The system according to claim 1,wherein the layout of the keyboard is a 2×3 grid and is based on QWERTY.7. The system according to claim 1, wherein the keyboard is disposedwithin a flexible printed circuit.
 8. A method of eyes-free text entry,comprising: receiving, by processing circuitry, a haptic input providedto a keyboard mounted on a finger of a user, the haptic input being anindication of an alphabetical letter determined based on a location ofthe haptic input on the keyboard; generating, by the processingcircuitry, a list of candidate words based on the received haptic input,each candidate word of the list of candidate words being associated witha probability thereof; displaying, by the processing circuitry, thegenerated list of candidate words to the user via a display of awearable device; receiving, by the processing circuitry, a selection ofa particular candidate word of the list of candidate words; andappending, by the processing circuitry, the particular candidate word ofthe list of candidate words corresponding to the received selection to apresent sentence structure, wherein the keyboard has a layout based on aspatial model reflecting spatial awareness, by the user, of keylocations on the finger.
 9. The method according to claim 8, furthercomprising receiving, by the processing circuitry, a corrective hapticinput to the keyboard indicating that a prior haptic input should beignored, the corrective haptic input being a swipe of a thumb of theuser.
 10. The method according to claim 8, further comprising receiving,by the processing circuitry, a directive haptic input to the keyboardindicating that a candidate word of the generated list of candidatewords is incorrect, the directive haptic input being a swipe of a thumbof the user.
 11. The method according to claim 8, further comprisingranking, by the processing circuitry, each candidate word of thegenerated list of candidate words based on a respective probability ofeach candidate word.
 12. The method according to claim 11, furthercomprising displaying, by the processing circuitry, a ranked candidateword of the generated list of candidate words to the user via thedisplay of the wearable device.
 13. The method according to claim 8,wherein the probability associated with each candidate word of thegenerated list of candidate words is calculated by generating, by theprocessing circuitry, a probability based on an application of thespatial model to the received haptic input, the spatial model describinga relationship between touch locations of the user and locations of keysof the keyboard, generating, by the processing circuitry, a probabilitybased on an application of a language model to the received hapticinput, the language model providing probability distributions of asequence of words for a given language, and combining, by the processingcircuitry, the generated probability based on the application of thespatial model and the generated probability based on the application ofthe language model to generate the probability associated with eachcandidate word of the list of candidate words.
 14. An apparatus foreyes-free text entry, comprising: processing circuitry configured toreceive a haptic input provided to a keyboard mounted on a finger of auser, the haptic input being an indication of an alphabetical letterdetermined based on a location of the haptic input on the keyboard;generate a list of candidate words based on the received haptic input,each candidate word of the list of candidate words being associated witha probability thereof; display the generated list of candidate words tothe user via a display of a wearable device; receive a selection of aparticular candidate word of the list of candidate words; and append theparticular candidate word of the list of candidate words correspondingto the received selection to a present sentence structure, wherein thekeyboard has a layout based on a spatial model reflecting spatialawareness, by the user, of key locations on the finger.
 15. Theapparatus according to claim 14, wherein the processing circuitry isconfigured to calculate the probability associated with each candidateword of the list of candidate words by generating a probability based onan application of the spatial model to the received haptic input, thespatial model describing a relationship between touch locations of theuser and locations of keys of the keyboard, generating a probabilitybased on an application of a language model to the received hapticinput, the language model providing probability distributions of asequence of words for a given language, and combining the generatedprobability based on the application of the spatial model and thegenerated probability based on the application of the language model togenerate the probability associated with each candidate word of the listof candidate words.
 16. The apparatus according to claim 14, wherein theprocessing circuitry is further configured to rank each candidate wordof the generated list of candidate words based on a respectiveprobability of each candidate word.
 17. The apparatus according to claim14, wherein the processing circuitry is further configured to receive acorrective haptic input to the keyboard indicating that a prior hapticinput should be ignored, the corrective haptic input being a swipe of athumb of the user.
 18. The apparatus according to claim 14, wherein theprocessing circuitry is further configured to receive a directive hapticinput to the keyboard indicating that a candidate word of the generatedlist of candidate words is incorrect, the directive haptic input being aswipe of a thumb of the user.
 19. The apparatus according to claim 14,wherein the layout of the keyboard is a 2×3 grid and is based on QWERTY.20. The apparatus according to claim 14, wherein the keyboard isdisposed within a flexible printed circuit.